Differentially private deep learning has recently witnessed advances in computational efficiency and privacy-utility trade-off. We explore whether further improvements along the two axes are possible and provide affirmative answers leveraging two instantiations of \emph{group-wise clipping}. To reduce the compute time overhead of private learning, we show that \emph{per-layer clipping}, where the gradient of each neural network layer is clipped separately, allows clipping to be performed in conjunction with backpropagation in differentially private optimization. This results in private learning that is as memory-efficient and almost as fast per training update as non-private learning for many workflows of interest. While per-layer clipping with constant thresholds tends to underperform standard flat clipping, per-layer clipping with adaptive thresholds matches or outperforms flat clipping under given training epoch constraints, hence attaining similar or better task performance within less wall time. To explore the limits of scaling (pretrained) models in differentially private deep learning, we privately fine-tune the 175 billion-parameter GPT-3. We bypass scaling challenges associated with clipping gradients that are distributed across multiple devices with \emph{per-device clipping} that clips the gradient of each model piece separately on its host device. Privately fine-tuning GPT-3 with per-device clipping achieves a task performance at $\epsilon=1$ better than what is attainable by non-privately fine-tuning the largest GPT-2 on a summarization task.

大型预审慎的模型可以私下微调以实现非私有模型的性能。这些结果中的一个共同主题是令人惊讶的观察结果，即高维模型可以实现有利的隐私性权衡。这似乎与差异私有凸学习的模型尺寸依赖性相矛盾，并提出了以下研究问题：差异私人学习的性能何时不会随着模型大小的增加而降低？我们确定投影到子空间上的梯度的幅度是决定性能的关键因素。为了确切地为私人凸学习的特征，我们引入了一个条件，即我们将限制Lipschitz的连续性限制并得出了在其他条件下与维度无关的过多经验和人口风险的界限。我们从经验上表明，在大型语言模型的私人微调中，在本地最佳距离附近评估的梯度主要由一些主要组件控制。这种行为类似于我们在凸面设置中获得尺寸独立界限的条件。我们的理论和经验结果共同为大规模私人微调成功提供了可能的解释。

私人随机梯度下降（DP-SGD）是私人深度学习最新进展的主力算法。它为数据集中的所有数据点提供了单个隐私保证。我们提出了一种有效的算法，以在释放由DP-SGD培训的模型时计算单个示例的隐私保证。我们使用算法来研究许多数据集中的个人隐私参数。我们发现，大多数示例比最严重的案例拥有更强的隐私保证。我们进一步发现，训练损失和示例的隐私参数是非常相关的。这意味着在模型效用方面服务不足的群体在隐私保证方面同时服务不足。例如，在CIFAR-10上，测试准确性最低的课程的平均$ \ epsilon $比班级的平均$ \ epsilon $高26.3％。我们还运行会员推理攻击，以表明这反映了不同的经验隐私风险。

我们为大规模训练的大规模训练语言模型提供了更简单，更稀疏，更快的算法，这些算法在许多标准的NLP任务上实现了最新的隐私与实用性权衡。我们为此问题提出了一个元框架，这是受高度参数效率方法进行微调成功的启发。我们的实验表明，这些方法的差异化适应能力在三个重要方面优于以前的私人算法：实用程序，隐私以及私人培训的计算和记忆成本。在许多经常研究的数据集中，私人模型的实用性接近了非私人模型的方法。例如，在MNLI数据集上，我们使用Roberta-large的准确度为87.8 \％$，使用Roberta-Base $ 83.5 \％$，其隐私预算为$ \ Epsilon = 6.7 $。相比之下，缺乏隐私限制，罗伯塔·莱格（Roberta-Large）的准确度为$ 90.2 \％$。我们的发现对于自然语言生成任务类似。与DART，GPT-2-SMALL，GPT-2中，GPT-2-MEDIUM，GPT-2-LARGE和GPT-2-XL的私人微调达到38.5、42.0、43.1和43.8（$ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ 43.8） epsilon = 6.8，\ delta = $ 1E-5），而非私人基线为$ 48.1 $。我们所有的实验都表明，较大的模型更适合私人微调：虽然众所周知，它们旨在非优先实现卓越的准确性，但我们发现当引入隐私时，它们也更好地保持其准确性。

培训深神经网络（DNNS）在企业和云数据中心都广受欢迎。现有的DNN培训调度程序将GPU视为主要资源，并分配其他资源，例如CPU和内存与作业要求的GPU数量成正比。不幸的是，这些调度程序不考虑作业对CPU，内存和存储资源分配的敏感性的影响。在这项工作中，我们提出了Synergy，这是一种对共享GPU群集的资源敏感调度程序。通过乐观的分析，协同作用侵犯了DNN对不同资源的敏感性；某些工作可能会从GPU育儿分配中受益更多，而某些工作可能不会受到GPU育儿分配的影响。 Synergy使用新的近乎最佳的在线算法在共享的多租户集群上安排的一组作业进行了多余的工作量感知作业。我们的实验表明，与传统的GPU育儿计划相比，工作量感知的CPU和内存分配可以提高平均JCT高达3.4倍。

Explainable Artificial Intelligence (AI) in the form of an interpretable and semiautomatic approach to stage grading ocular pathologies such as Diabetic retinopathy, Hypertensive retinopathy, and other retinopathies on the backdrop of major systemic diseases. The experimental study aims to evaluate an explainable staged grading process without using deep Convolutional Neural Networks (CNNs) directly. Many current CNN-based deep neural networks used for diagnosing retinal disorders might have appreciable performance but fail to pinpoint the basis driving their decisions. To improve these decisions' transparency, we have proposed a clinician-in-the-loop assisted intelligent workflow that performs a retinal vascular assessment on the fundus images to derive quantifiable and descriptive parameters. The retinal vessel parameters meta-data serve as hyper-parameters for better interpretation and explainability of decisions. The semiautomatic methodology aims to have a federated approach to AI in healthcare applications with more inputs and interpretations from clinicians. The baseline process involved in the machine learning pipeline through image processing techniques for optic disc detection, vessel segmentation, and arteriole/venule identification.

In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composing the solutions of previously solved primitive tasks (task composition). Otherwise, prior knowledge can be used to adjust the reward function for a new problem, in a way that leaves the optimal policy unchanged but enables quicker learning (reward shaping). In this work, we develop a general framework for reward shaping and task composition in entropy-regularized RL. To do so, we derive an exact relation connecting the optimal soft value functions for two entropy-regularized RL problems with different reward functions and dynamics. We show how the derived relation leads to a general result for reward shaping in entropy-regularized RL. We then generalize this approach to derive an exact relation connecting optimal value functions for the composition of multiple tasks in entropy-regularized RL. We validate these theoretical contributions with experiments showing that reward shaping and task composition lead to faster learning in various settings.

Causal phenomena associated with rare events frequently occur across a wide range of engineering and mathematical problems, such as risk-sensitive safety analysis, accident analysis and prevention, and extreme value theory. However, current methods for causal discovery are often unable to uncover causal links between random variables that manifest only when the variables first experience low-probability realizations. To address this issue, we introduce a novel algorithm that performs statistical independence tests on data collected from time-invariant dynamical systems in which rare but consequential events occur. We seek to understand if the state of the dynamical system causally affects the likelihood of the rare event. In particular, we exploit the time-invariance of the underlying data to superimpose the occurrences of rare events, thus creating a new dataset, with rare events are better represented, on which conditional independence tests can be more efficiently performed. We provide non-asymptotic bounds for the consistency of our algorithm, and validate the performance of our algorithm across various simulated scenarios, with applications to traffic accidents.

Chest X-ray (CXR) datasets hosted on Kaggle, though useful from a data science competition standpoint, have limited utility in clinical use because of their narrow focus on diagnosing one specific disease. In real-world clinical use, multiple diseases need to be considered since they can co-exist in the same patient. In this work, we demonstrate how federated learning (FL) can be used to make these toy CXR datasets from Kaggle clinically useful. Specifically, we train a single FL classification model (`global`) using two separate CXR datasets -- one annotated for presence of pneumonia and the other for presence of pneumothorax (two common and life-threatening conditions) -- capable of diagnosing both. We compare the performance of the global FL model with models trained separately on both datasets (`baseline`) for two different model architectures. On a standard, naive 3-layer CNN architecture, the global FL model achieved AUROC of 0.84 and 0.81 for pneumonia and pneumothorax, respectively, compared to 0.85 and 0.82, respectively, for both baseline models (p>0.05). Similarly, on a pretrained DenseNet121 architecture, the global FL model achieved AUROC of 0.88 and 0.91 for pneumonia and pneumothorax, respectively, compared to 0.89 and 0.91, respectively, for both baseline models (p>0.05). Our results suggest that FL can be used to create global `meta` models to make toy datasets from Kaggle clinically useful, a step forward towards bridging the gap from bench to bedside.

Tuberculosis (TB), an infectious bacterial disease, is a significant cause of death, especially in low-income countries, with an estimated ten million new cases reported globally in $2020$. While TB is treatable, non-adherence to the medication regimen is a significant cause of morbidity and mortality. Thus, proactively identifying patients at risk of dropping off their medication regimen enables corrective measures to mitigate adverse outcomes. Using a proxy measure of extreme non-adherence and a dataset of nearly $700,000$ patients from four states in India, we formulate and solve the machine learning (ML) problem of early prediction of non-adherence based on a custom rank-based metric. We train ML models and evaluate against baselines, achieving a $\sim 100\%$ lift over rule-based baselines and $\sim 214\%$ over a random classifier, taking into account country-wide large-scale future deployment. We deal with various issues in the process, including data quality, high-cardinality categorical data, low target prevalence, distribution shift, variation across cohorts, algorithmic fairness, and the need for robustness and explainability. Our findings indicate that risk stratification of non-adherent patients is a viable, deployable-at-scale ML solution.